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RATIONAL MODE LOCKING FOR HOMEOMORPHISMS OF THE 2-TORUS 
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Abstract. In this paper we consider homeomorphisms of the torus M 2 /Z 2 , homotopic to the identity, 
and their rotation sets. Let / be such a homeomorphism, / : R 2 —> R 2 be a fixed lift and p(f) C M 2 be 
its rotation set, which we assume to have interior. We also assume that the frontier of p(/) contains 
a rational vector p E Q 2 and we want to understand how stable this situation is. To be more precise, 
we want to know if it is possible to find two different homeomorphisms f± and / 2 , arbitrarily small 
C°-perturbations of /, in a way that p does not belong to the rotation set of fi but belongs to the 
interior of the rotation set of / 2 , where f\ and /2 are the lifts of fi and /2 that are close to /. We give 
two examples where this happens, supposing p = (0,0). The first one is a smooth diffeomorphism 
with a unique fixed point lifted to a fixed point of /. The second one is an area preserving version 
of the first one, but in this conservative setting we only obtain a C° example. We also present two 
theorems in the opposite direction. The first one says that if / is area preserving and analytic, we 
cannot find fi and /2 as above. The second result, more technical, implies that the same statement 
holds if / belongs to a generic one parameter family ( ft)te[o,i] °f C 2 -diffeomorphisms of T 2 (in the 
sense of Brunovsky). In particular, lifting our family to a family ( ft)te[o,i ] °f plane diffeomorphisms, 
one deduces that if there exists a rational vector p and a parameter t* E (0,1) such that p( ft *) has 
non empty interior, and p 0 p(ft) for t < t* close to £*, then p 0 int(p(/t)) for all t > t* close to £*. 
This kind of result reveals some sort of local stability of the rotation set near rational vectors of its 
boundary. 


1. Introduction and main results 

The main motivation for this paper is to study how the rotation set of a homeomorphism of the two 
dimensional torus T 2 , homotopic to the identity, changes as the homeomorphism changes. For instance, 
suppose we consider a one parameter continuous family ft : T 2 — > T 2 of such maps, a continuous family 
of lifts ft : M 2 —> M 2 and want to study how the parameterized family of rotation sets t \—> p(ft) varies. 
The rotation set is a non empty compact convex subset of the plane (see definition below), which varies 
continuously with the homeomorphism, at least in the special situation when it has interior (see [ED- 
iii particular, we are interested in the following problem: Suppose / : T 2 — ► T 2 is a homeomorphism 
homotopic to the identity and its rotation set for a given lift /, which is supposed to have interior, has 
a point p in its boundary with both coordinates rational. Is it possible to find two different arbitrarily 
small C°-perturbations of /, denoted /i and fi , in a way that p does not belong to the rotation set of 
the lift of fi close to / but is contained in the interior of the rotation set of the lift of fi close to /? 
In other words we are asking if the rational mode locking found by de Carvalho, Boyland and Hall [3] 
in their particular family of homeomorphisms is, in a certain sense, a general phenomena or not. Our 
main theorems and examples will show that the answer to this question depends on the hypotheses 
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we have. In general, we can find such maps /i and fi as described above, but if we assume certain 
hypotheses on /, then this sort of “local mode locking” happens. 

Even in the much simpler context of orientation preserving circle homeomorphisms, the only maps with 
rational rotation number which can be perturbed in an arbitrarily C'°-small way in order to decrease 
or increase their rotation numbers (the analogous one dimensional version of our condition) are the 
ones conjugate to rational rotations. In the context of degree one circle endomorphisms, if / : T 1 —> T 1 
is such an endomorphism, / : R —> R is a fixed lift and p{f) is its rotation set (which in this case is a 
compact interval), if we assume that the rotation set is not reduced to a point and has a rational end 
p, then it is not possible to find C° neighbors of / in the space of lifts, one whose rotation set does 
not contain p and the other with p in the interior of its rotation set (see Theorem [5] below). 

In order to make things precise and to present our main results, a few definitions are necessary: 

- We denote T 2 = R 2 /Z 2 the flat torus and n : R 2 — > T 2 the universal covering projection. 

- We denote Diffo(T 2 ) the space of C r (for r = 0, 1, 2,..., oo, w) diffeomorphisms (homeomorphisms if 
r = 0) of the torus homotopic to the identity and Diff 0 (T 2 ) the space of lifts of elements of Diffg(T 2 ) 
to the plane. 


- We write p\ : (x, y) K > x and P2 : {x, y) i—>■ y for the standard projections defined on the plane. 

- Given / £ Diffg(T 2 ) and a lift / £ Diff 0 (T 2 ) the rotation set p(f) of / can be defined following 
Misiurewicz and Ziemian [12] as: 


p(f )=n u 

i^l n^i 



z G 


R 2 


This set is a compact convex subset of R 2 (see [12]), and it was proved in [8] and |T2] that all points in 
its interior are realized by compact /-invariant subsets of T 2 , which can be chosen as periodic orbits 
in the rational case. By saying that some vector p £ p(f) is realized by a compact /-invariant set, 
we mean that there exists a compact /-invariant subset I\ C T 2 such that for all z £ K and any 
z £ 7r -1 ({z}), one has 


r(z)-z 

lim - = p. 

n—too ft 

Moreover, the above limit, whenever it exists, is called the rotation vector of z and denoted p{z). 


As the rotation set is a compact convex subset of the plane, there are three possibilities for its shape: 
it is a point, a linear segment or it has interior. In this paper we only consider the situation when the 
rotation set has interior. The first main result is: 


Theorem 1. Let f £ Diff 0 (T 2 ) be an area preserving diffeomorphism such that the interior of p(f) 
is non empty and its frontier contains a rational vector p £ Q 2 . Then, one of the following situations 
occurs: 

- there exists a neighborhood U of f in Diff 0 (T 2 ) such that p £ p(f') for every f £ U; 

- there exists a neighborhood U of f in Diff 0 (T 2 ) such that p int (p(f)) for every f £U. 


The next result permits to understand the behaviour of generic one parameters families. Fix p = 
( p/q,r/q ) £ Q 2 , where q ^ 1 and g.c.d(p, q, r) = 1. Suppose that / £ Diff 0 (T 2 ) can be approximated 
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arbitrarily close by f £ Diff 0 (T 2 ) such that p £ int(p(/')). One deduces that the fixed point set of 
f q — ( p , r) is not empty. Suppose moreover that this set is discrete (which means that it projects 
onto a finite set of T 2 ) and that / may be approximated arbitrarily close by f" £ Diff 0 (T 2 ) such that 
p p(S")- I n that case, the Lefschetz index of every fixed point z of f q — {[p,r ) is equal to 0. In 
particular, if / £ Diff 0 (T 2 ), one knows that 1 is an eigenvalue of Df q (fz). 

Let us say that a diffeomorphism / £ Diff^T 2 ) is not highly degenerate at p, if the fixed point set of 
f q — ( p,r) is discrete and if for every point z in this set, Df q (fz) has at least one eigenvalue different 
from 1. 

Theorem 2. Let f £ Diff 0 (T 2 ) be such that the interior of p(f) is non empty and its frontier contains 
a rational vector p £ Q 2 . Suppose moreover that f is not highly degenerate at p. Then, one of the 
following situations occurs: 

- there exists a neighborhood U of f in Diff 0 (T 2 ) such that p £ p(f') for every f £ U; 

- there exists a neighborhood U of f in Diff 0 (T 2 ) such that p £ int(p(/')) for every f eU. 

This result implies the statement about generic families explained in the abstract. More precisely, for 
C r -generic one parameter families of diffeomorphisms, r ^ 1, Brunovski has shown, see [4], that the 
only bifurcations that create or destroy periodic points are saddle-nodes and period doubling. Theorem 
[2] hypotheses imply that the creation of the periodic orbits with rotation vector p = ( p/q,r/q ) had 
to be through a saddle-node type of bifurcation because the map / has neighbors without g-periodic 
points with rotation vector equal to p. 

The next two results indicate the requirment of the hypotheses in the previous theorems. 

— - OO _ 

Theorem 3. There exists f £ Diff 0 (T 2 ) such 

- for every neighborhood U of f in Diff 0 (T 2 ) there exists f £ U such that (0,0) ^ /?(/'); 

- for every neighborhood U of f in Diff 0 (T 2 ) there exists f £lA such that (0, 0) £ int (p{f)); 

- f has a unique fixed point, up to translation by a vector of Z 2 . 

The example in Theorem [3] is not area preserving. We can construct an area preserving example, but 
it will not be differentiable 

~ -0 

Theorem 4. There exists f £ Diff 0 (T 2 ) such 

- for every neighborhood U of f in Diff 0 (T 2 ) there exists f GlA such that (0, 0) ^ p(f'); 

- for every neighborhood U of f in Diff 0 (T 2 ) there exists f &IA such that (0, 0) £ int (p(f)); 

- f has a unique fixed point, up to translation by a vector of Z 2 ; 

- f is area preserving. 

To conclude, we present a simple theorem about endomorphisms of the circle, which shows that in this 
situation, the conclusion of Theorem [l] holds in full generality. The proof we present is due to Andres 
Koropecki. Write 7r : R —> T for the universal covering projection of T = R/Z. Denote Endo(T 1 ) 
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the space of continuous maps / : T —> T homotopic to the identity and Endo(T) the space of lifts of 
elements of Endo(T) to the line. The rotation set p(f) of / £ EndoT) is a real segment defined as: 

i^l n^i v 

Let / £ Endo(T) be such that p(f) is not reduced to a point and its frontier contains a rational number 
p. We will prove that one of the following situations occurs: 

- there exists a neighborhood U of / in Endo(T) such that p £ p(f') for every /' £ ZY; 

- there exists a neighborhood U of / in Endo(T) such that p qL int(p(f')) for every f £ U. 

In fact we have the stronger following result: 

Theorem 5 . Let p = p/q be a rational number and f £ Endo(T) such that f q —p^f= Id. Then, one of 
the following situations occurs: 

- there exists a neighborhood U of f in Endo(T) such that p £ p(f') for every f £ U; 

- there exists a neighborhood U of f in Endo(T) such that p ^ int(p(/')) for every f £ U. 

Proof. If the map f q — p — Id takes a positive and a negative value, it will be the same for f' q — p — Id if 
f £ Endo(T) is close to /. This implies that f' q — p has at least one fixed point, so the first assertion 
is true. If f q — p — Id does not vanish, it will be the same for f' q — p — Id if f £ Endo(T) is close to 
/. This implies that p does not belong to p(f ), so the second assertion is true. As we suppose that 
f q — p — Id is not identically zero, it remains to study the case where it vanishes but has constant 
sign. Suppose that there exists Xq such that g(x o) < xq, where g = f q — p. By hypothesis g(x) ^ x 
for every x £ R and so g([x o — l,a;o]) C (—00, £0). We deduce that there exists a neighborhood U 
of / in End 0 (T) such that g'{[x 0 — l,£o]) C (—00, x 0 ), where g' = f' q — p. Consequently, one gets 
</((—00, Xq]) C (— 00, Xq) and more generally g ,n ((— 00, Xo]) C (— 00, xq) for every n ^ 1 . The fact that 

x ^ Xo and n ^ 1 => g n (x ) ^ Xg 

implies that the rotation set of f in included in (— 00,p/q]. Similarly, if f q — p — Id vanishes and is 
non negative, there exists a neighborhood U of / in End 0 (T) such that p(f') C \p/q, +00). □ 

This paper is organized as follows. In the next section we present a brief summary on the local 
dynamics near fixed points of area preserving analytic plane diffeomorphisms. In the third section we 
prove a fundamental result necessary to get Theorems 1 and 2 . The precise proofs of Theorems 1 , 2 , 3 
and 4 will be given in the fourth section. 

2 . LOCAL STUDY OF ANALYTIC AND AREA-PRESERVING PLANAR DIFFEOMORPHISMS 

The dynamics near isolated singularities of analytic vector fields in the plane is very well understood, 
at least when the topological index of the singularity is not 1 (see for instance Dumortier 0 )- It can 
be proved that, if the singularity is neither a focus or a center (which have topological index equal 
1 ), then the dynamics near it can be obtained from a finite number of sectors, glued in an adequate 
way. Topologically, these sectors can be classified in four types: elliptic, hyperbolic, expanding and 
attracting. Dumortier, Rodrigues and Roussarie studied this problem for planar diffeomorphisms near 
fixed points in jTj. Let us recall a fundamental notion in their study: a smooth planar vector field X , 
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vanishing at the origin has Lojasiewicz type of order k ^ 1 if there exist C > 0 and 5 > 0 such that 
||x|| < S => ||A(:r)|| ^ C'||a;|| fe . It can be easily seen that this notion depends only on the /c-jet J\ at 
(0,0). In particular X has Lojasiewicz type of order fc > 1 if and only if it is the case for J\ and one 
can extend this notion to any formal vector field by looking at the truncated series of order k. If X is 
a real analytic planar vector field vanishing at the origin, it has Lojasiewicz type, which means that 
there exists k ^ 1 such that A' has Lojasiewicz type of order k ^ 1, (see Bierstone-Milman W- 
The situation we want to understand in this sub-section is the following: Is there a topological picture 
of the dynamics near an index zero isolated fixed point of an analytic area-preserving planar diffeo- 
morphism? It turns out that the area-preservation together with the zero index hypothesis imply that 
the eigenvalues of the derivative of the diffeomorphism at the fixed point are both equal to 1. We will 
suppose that the fixed point is the origin. The area-preservation implies that the infinite jet J/° of / 
at (0,0) is the time one mapping of a unique formal vector field X (defined by a formal series), see 
[T5] and Moser Q2j. Let us fix a smooth vector field A' such that Jjj? = A and consider the time one 
map g of the flow defined by X. Let us prove by contradiction that A has Lojasiewicz type. If not, 
by Proposition 2 of Llibre-Saghin m , one can choose X such that 0 is a non isolated zero of X and 
consequently a non isolated fixed point of g. One deduces that J|f. Id has not Lojasiewitz type. On 
the other hand, J/l Id has Lojasiewitz type because / is analytic. So, we have a contradiction because 
/ and g have the same infinite jet at (0,0). 

Let us explain now why X and / — Id have the same index at 0. The fact that / — Id and g — Id 
have the same index (which is equal to zero) at the origin is given by Proposition 1 of {PS]- It is a 
consequence of the fact that these vector fields have the same infinite jet at 0 and that this jet has 
Lojasiewicz type. Write g t , t £ (0,1], for the time t map of the flow induced by X. There exists S > 0 
such that the vector field X and the vector fields g t — Id, t £ (0,1] have no zero satisfying 0 < ||x|| ^ 6. 
Otherwise, g = g i has an invariant curve in any neighborhood of 0, which implies that the index of 
g — Id is 1, in contradiction with the hypothesis. By computing the indices on the circle of equation 
||a;|| = <5, one deduces that the indices of the g t — Id are all the same. By letting t tend to 0, one 
deduces that this common index is the index of X. 

The fact that the index of X is not 1 implies, by [5], that A has at least one characteristic orbit at 0, 
which means an integral curve 7 of A' or —X defined on [0, 00) such that: 


0, lim 7(0 = 0, lim 7(0/117(011 exists. 

c—>■+OO t—»■+OO 


The existence of a characteristic orbit only depends on a finite jet of X and is independent of the choice 
of X. The fact that X has Lojasiewicz type and has a characteristic orbit permits us to apply [Tj, 
Theorem D: there exists a vector field X' such that Jff, = X and such that / is weakly-C°-conjugated 
to the time-1 map of the flow induced by A'. 

The above theorem implies, see J7j pages 39-40, that the dynamics of / in a neighborhood of the 
origin is obtained by gluing a finite number of sectors, which can be attracting, expanding, elliptic or 
hyperbolic and moreover, in our situation, as we are supposing that / preserves the area, there can 
not be elliptic, expanding and attracting sectors. As the topological index of the fixed point is 0, there 
must be exactly two invariant hyperbolic sectors and the dynamics is topologically as in Figure 1. 
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Figure 1. 

3. Fundamental proposition 

We introduce in this section an important notion for our problem. Let / be a homeomorphism of 
a surface M. A fixed point zq of / will be called trivializable if there exists a continuous chart 
h : U C M —> V C M 2 at z 0 such for every z £ U D / -1 (t/), one has p\ o h o f(z) > p\ o h(z) if z Zq. 
Let us give two categories of trivializable fixed points. 

Proposition 6. Let f be a C 2 -cLiffeomorphism of a surface M and zq a fixed point that is a saddle-node, 
meaning that one of the eigenvalue of Df(zo) is 1 and the other one is not. Then Zq is trivializable. 

Proof. By considering / -1 if necessary, we can assume that det(-D/(zo)) < 1. This proposition follows 
from the following lemma (for example, see Carr 0, Theorem 1, page 16 and Lemma 1, page 20, where 
the results are proved for vector fields, analogous proofs holding for maps, as stated in pages 33-35): 

Lemma 7. Assume f : R 2 — > R 2 is a C 2 -diffeomorphism which fixes the origin and Df( 0,0) has 1 and 
A £ (0,1) as eigenvalues. If we write f in coordinates such that f(x,y) = (x + u(x, y), Xy + v(x, y)), 
where the functions u, v and their first derivatives vanish at the origin, then there exists a C 2 -function 
h defined for |a"| sufficiently small such that h( 0) = h'{ 0) = 0, whose gmph is invariant under iterates 
of f, a neighborhood U of (0,0) and C > 0 such that for any segment of orbit (xk, yk)o^k^n included 
in U, one has \y n — h(x n )\ ^ C.A n . |j/o — h(x o)|. 

As we are assuming that the origin is an isolated fixed point, which is a saddle-node, without loss of 
generality we can suppose that points in the center manifold with negative x coordinate converge to 
the origin under positive iterates of / and points in the center manifold, with positive x coordinate 
converge to the origin under negative iterates of /. 

So, from Lemma [7] the dynamics in a neighborhood of the origin can be obtained by gluing exactly 
three sectors, two adjacent hyperbolic sectors and one attracting one. Thus, under a C°-coordinate 
change, it is easy to see that the fixed point is trivializable: the vertical foliation in this system of 
coordinates is topologically transverse to the natural foliation by locally invariant leaves defined by 
the sadlle-node (see Figure 2). 

□ 

Proposition 8. Let f be an area preserving and analytic diffeomorphism of a surface M. Every 
isolated fixed point zq of f of Lefschetz index 0 is trivializable. 
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Proof. The proposition is an immediate consequence of the results stated in the previous section. 
Note that in this case, there exists a continuous chart h : U C M —> V C R 2 at Zo such for every 
z £ U n / _ 1 (C/), one has p\oho f(z ) > p\ o h(z) and P 2 ° h ° f(z) = p- 2 ° h(z) if 2 / Zq. □ 

Let us state now the fundamental proposition: 

Proposition 9. Suppose that f £ Diffg(T 2 ) has only trivializable fixed points. Then, there exists a 
neighborhood U of f in £ Diffg(T 2 ) such that (0,0) ^ int(p(/')) for every f £ U. 

Proof. By hypothesis, fix(/) is discrete and projects onto a finite set of T 2 . Fix £0 > 0 and set 
Ro(z) = z + [—£o,£o] 2 for every z £ fbc(/). Conjugating / in Diffg(T 2 ) if necessary and choosing 
£0 > 0 small enough, one can suppose that the rectangles Rq(z), z £ fix(/), are pairwise disjoint and 
that for every z £ fix(/) and every z' £ Rq(z) \ {z}, one has pi o / _ 1 (F) < Pi(z') < p\ o f{z'). 

Fix 0 < e\ < £1 < £0 and set 

R o(z) = z + ( f D £ o) x (-£ 0 ,£ 0 ), 

R o(z) = z + (- £ o, -£1) x (- £ o, £ o), 

R i{z) = z+ [-£[,£[] x [—£ 1 ,£ 1 ], 

R l{z) = Z+ [ £\, £\] x (-£!,£!). 


R(z) 


R(z) 



R ( z) 


Figure 3. 
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We will suppose ei small enough to ensure that for every z £ hx(/), one has 

f(z+ [ei, £i] 2 ) U f~ 1 (z + [e± , £i] 2 ) C int(i? 0 (5)). 

This implies that if e\ is small enough then for every z £ fix(/) one has 

z 1 £ Ri(z) and f(ft) ?RUz)^ f(z') £ R+(z) 

and 

S' £ Rftz) and <£ R\{z) => f~\ft) £ Rf{z). 

We will suppose that this is the case. 

Now fix e 2 £ (0, £ 1 ) and set Ro(z) = z+ [— £ 2 ,£ 2 ] 2 - We will suppose £2 small enough to ensure that 
for every z £ fix(/) one has 

f(R 2 (z)) u f- 1 (R 2 (z)) c int(i?!(2)). 

We will say that /' £ Diff^T 2 ) is a positive perturbation of / if 

(1) fix(f) C U£gfi x (7) int(i? 2 (2)); 

(2) for all z £ fix(/) and z’ £ Rq(z) \ int(i? 2 (^)), the following inequality holds 

Pi 0 /'^(X) < Pii?) <Pi° 

(3) for all z £ hx(/) and zl £ R\(z) such that ft {ft') fL R\(z ), then ft {ft') £ 

(4) for all S' £ hx(/) and z! £ Rift) such that f~ l ft') ^ then ft -1 ft') £ Rffft)\ 

(5) for every S' £ hx(/), one has f'(R 2 ft)) U / ,- 1 (i? 2 (<? 2 )) C int(i?i(S)). 

Note that the set U of positive perturbations of / is an open neighborhood of / in Diff^T 2 ). Of course, 
it contains /. The fact that it is open follows from the fact that the properties (1), (2) and (5) are 

open, and that the set of maps satisfying (2), (3) and (5) is open, as is the set of maps satisfying (2), 

(4) and (5). 

To get the proposition, we will prove that that (0, 0) ^ int(p(/')) if f is a positive perturbation of /. 

We will argue by contradiction, supposing that (0,0) £ int(p(/')), for a positive perturbation f of 
/. In that case, using a result of Franks (5], one knows that there exists three periodic orbits Oi, 
1 ^ ^ 3, of the homeomorphism /' of T 2 lifted by f such that (0, 0) belongs to the interior of the 

convex hull of the p(Oi), 1 < * < 3. For every 2 £ 7 r(hx(/)), we set Rq(z) = tt(R 0 (z)) if z = n(z), and 
define similarly Rq (z), Rq(z), Ri(z), R\{z) and R 2 {z). 

Lemma 10. If the orbit Oi meets a rectangle Rift), z £ 7r(fix(/)), then there exists a positive 
perturbation f" of f such that O 2 and O 3 are periodic orbits of the homeomorphism f" of T 2 lifted by 
f", with unchanged rotation vectors and a periodic orbit 0\ C Oi whose rotation vector is a multiple 
of p(0 1 ) by a factor larger than 1, such that 

• tt(Oi n Rift)) < tHOi n Rift)) 

• 0[ n Rift') = Oi PI Rift') for every z' £ 7 r(hx(/)) \ {z}. 

Proof. Suppose that z £ Oi n Rift). The fact that the rotation vector of 0\ does not vanish implies 
that there exist k~ < 0 < k + such that 

f k ~ft[) # Rftz 0 ), f' k+ ft[) # Rftzo) ft k ft[) £ Rftzo) if k~ <k< k+. 

One deduces that f' k ft) £ Rq{z) and f' k ( z ) £ Rq ft) because ft is a positive perturbation of /. One 
can find a horizontal graph F in int(i?o(^)) \ Rift) joining f k (z) to f k+ ft), which means a simple 
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path that projects injectively onto the first factor of T 2 , then a neighborhood U C int(Ro(z))\Ri(z) of 
T. The graph and the neighborhood can be chosen to meet 0\ U 0 2 U 0 3 only at the points f k (z) and 
f' k (z). One can find a homeomorphism h supported on U such that h(f' k (z)) = f' k (z). Moreover 
h may be chosen such that it is lifted to a homeomorphism h G Diffg(T 2 ) supported on 7r _1 (I/) and 
satisfying p 3 o h(z') > Pi(z') for every z' € R 2 . 


• r 



Figure 4. 

Let us explain why /" = h o f is a positive pertubation of / by verifying that the five properties of 
the definition of positive perturbation are satisfied by f" 

(2) If there exists z G fix(/) such that z' G Ro(z) \ int(I? 2 ( 2 )), then h~ x (z') G Ro(z) \ int(i?2(^)) and 
so 

Pi ° /' _1 o h -1 ^') < pi o~h-\z') < Pi{z') <piof (: z') ^p\olioj' (z'). 

( 1 ) The maps f"~ l and /' _1 coincide on the complement of Ujefi x (/) Ro(z) and the last one has no 
fixed point in this complement, so the fixed point set of f" is included in Rq{z). The map f" 

satisfying 2 , its fixed point set is included in U?£fi x (/) int(I? 2 ( 2 ))- 

( 3 ) If there exists z G hx(/) such that J! G R\(z) and f"(z') R\(z), then f(z') £ R\(z) (otherwise 

f”(z') = f(z')) and so f'iz') G Rq (z), which implies that f”(z') G Rq (z), because h(R 3 (z)) C Rq (z). 

(4) If there exists z G fix(/) such that z' G Ri(z) and / ,,_ 1 (F) ^ -R*(5), then /' _ 1 (z') = /" _1 (? / ) ^ 
R*i(S), and so ]"~ x (z') = f" 1 ^) G R^(z). 

(5) For every z G fix(/), one has f'{R2(z)) U 1 (-^ 2 (^ 2 )) C int(i?i(i)). It implies that /"(i? 2 ( 5 )) = 
f(R 2 {z)) and f"~ 1 {R 2 {z2)) = f _1 (^ 2 (^ 2 )) and consequently that f"(R 2 (z)) U f”~ 1 (R 2 (z2)) C 
int(i?i (z))- 

Observe now that /' and f" coincide on 0 2 and 0 3 and that /' and /" coincide on 7r _ 1 (0 2 ) and 
7r _1 (0 3 ). So, O 2 and O 3 are periodic orbits of f" with rotation vectors unchanged. The orbit 0\ has 
been replaced by a periodic orbit 0 [ C 0 1 of shorter period whose rotation vector (for the lift /") is 
a multiple of the rotation vector of 0 \ (for /') by a factor larger than 1 . □ 

Note that (0,0) is still in the interior of the convex hull of the new rotation vectors, the old ones 
multiplied by numbers greater than 1. Applying the lemma finitely many times to each orbit Oi and 
each rectangle R\{z), one can always suppose that the orbits O, of /' do not meet the rectangles R\(z), 
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2 £ 7r(fix(/)). One can find h £ Diff[J(T 2 ) supported on U~ e fi x (/) Ri(z)> that satisfies p\oh(z') ^ Pi(z') 
for every z' £ K 2 and such that h o f(R 2 (z )) fl R 2 (z) = 0 for every z £ fix(/). 

Note that g = h o /' is fixed point free. Indeed: 

• p _1 and / ,_1 coincide on the complement of Ujyfixf/) ^ l>l ^ anc ^ / /_1 ^ as 110 ^ xe< ^ point i n 
this complement, so it is the same for <? _1 ; 

• If there exists z £ hx(/) such that !? £ Ri(z) \ int(i?2(2)), then p\o ho f(z') ^ p\ o f(z') > 

pi{z r y, 

• g(R 2 (z)) 0 R 2 (z) = 0 for every z £ hx(/). 

On the other hand, each Oi is a periodic orbit of the homeomorphism g of T 2 lifted by g , with 
unchanged rotation vector because C/ is disjoint from Ujefi x (/) -^i(z)- particular (0,0) belongs to 
the interior of p(p). This contradicts Franks’ result. □ 

4. Proofs of the theorems 

4.1. Proof of Theorem 1. Set p = ( Pi/q,p 2 /q ), where pi, p 2 and q are relatively prime. Replacing 
/ by f q ~ (pi)P 2 )) it is sufficient to study the case where p = (0,0). Denote / the homeomorphism of 
T 2 lifted by /. 

The function 

g:zi->(pio ]{z) -pi{z)) 2 + {p 2 oj{z)-p 2 (z)) 2 

lifts an analytic function g on T 2 that vanishes exactly on 7r(fix(/)). If this set is not finite, it contains 
a simple closed curve (see [T] or El). Such a curve must be homotopically trivial because the rotation 
set of / has interior. This means that hx(/) contains a simple closed curve T. This curves bounds 
a topological open disk D invariant by /. Moreover / is not the identity on this disk because it is 
analytic and not equal to the identity on the whole plane (its rotation set is not trivial). The fact 
that /|d is area preserving implies, by a classical consequence of Brouwer’s theory (see Franks [9] for 
example) that there exists a closed curve C C D such that the Lefschetz index of / on C is 1. Such 
a property, and consequently the existence of a fixed point is still satisfied for all C° perturbations of 
/. In particular, there exists a neighborhood U of / in Diffg(T 2 ) such that every /' £ U has a fixed 
point and consequently one has (0,0) £ p{f)- Suppose now that 7r(fix(/)) is finite. If the Lefchetz 
index of one fixed point of / is non zero, then there exists a neighborhood U of / in Diff^T 2 ) such 
that every f £U has a fixed point and consequently one has (0,0) £ p{f)- It remains to study the 
case where all indices are equal to zero. Proposition [8] tells us that every fixed point is trivializable. 
Applying Proposition [ 9 J one deduces that there exists a neighborhood U of / in £ Diff^T 2 ) such that 
(0,0) ^ int(p(/ / )) for all f £ U. 

4.2. Proof of Theorem 2. Suppose that / is not highly degenerate at p = ( p/q,r/q ) and that there 
exists a periodic point z of period q and rotation vector p such that 1 is not an eigenvalue of Df q (z). 
In that case the Lefschetz index of f q at z is different from 0 and one can conclude that there exists a 
neighborhood U of / in Diff 0 (T 2 ) such that p £ p(f') for every /' £ U. If 1 is an eigenvalue of Df q (z), 
for every point 2 of period q and rotation vector p, then every such point is trivializable by Propoistion 
[bj Applying PropositionjoJ one deduces that there exists a neighborhood U oi f in Diff 0 (T 2 ) such that 
p ^ int(p(/')) for every f £ U. 
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4.3. Proof of Theorem 3. The main properties of the map f G Diff 0 (T 2 ) that we want to construct 
are the following ones: 

• the rotation set of / is included in the non negative cone [ 0 , +oo ) 2 and contains the vectors 
( 0 , 0 ), ( 0 , 1 ) and ( 1 , 0 ); 

• each vector ( 0 , 0 ), ( 0 , 1 ) and ( 1 , 0 ) is the rotation vector of a fixed point of the diffeomorphism 
/ G Diff“(T 2 ) lifted by /; 

• the unique fixed point of rotation vector (0, 0) is (0,0) + Z 2 , it has a homoclinic point lifted to 
a heteroclinic point from ( 0 , 1 ) to ( 0 , 0 ) and a homoclinic point lifted to a heteroclinic point 
from ( 1 , 0 ) to ( 0 , 0 ); 

• the vector field z i-A f(z) — z has no values in the negative cone (—oo, 0 ) 2 ; 

• each vertical line {1} x I, k G Z, is sent on its right by / and each horizontall line R x {k} 
sent above. 

The third assertion allows us to perturb / in a way that the new map has a rotation set whose interior 
contains ( 0 , 0 ) and the last two assertions allow us to perturb f in a way that the new map has a 
rotation set which does not contain ( 0 , 0 ). 

The sets 

j(s,2/) G R 2 I \y\ < -^|sin( 7 ra;)| j , j(z,y) G R 2 | \x\ < ^ | sin( 7 ry) | j 

project by 7 r onto connected compact subsets of T 2 respectively denoted H and V. We set C = 
T 2 \ (H UP) and then define 

H = 7 t~ 1 (H), V = 7 T _ 1 (P), C = n~ 1 (C). 

— - 00 

Proposition 11. There exists fn G Diff 0 (T 2 ) such that: 

(1) the fixed point set of fn is H; 

(2) for every z G V U C, one has pi o fn(z) ^ Pi(z); 

(3) for every z G 1111(17), one has p\ o f H {z) > Pi(z) and P 2 ° fii(z) = P 2 (z)', 

(4) there exists zo G V such that 

lim = (0,1), lim f^(z 0 ) = (0,0); 

K—f —OO fc—>- + 00 

(5) there exists SiGtx {1/2} such that fii(z\) =zi + (1,0). 

Proof. Let us begin by choosing a pair of smooth Z 2 -periodic real valued functions £ and p on R 2 such 
that: 

• £ vanishes on ff U (Z x R) and is positive elsewhere; 

• 77 vanishes onffUC and is negative elsewhere (which means on int(P)). 

The map f e : z 1 —► z + e.(^(2), rj(z)) is smooth and lifts a smooth transformation of T 2 homotopic to 
the identity. As the set of C°° diffeomorphisms of the torus is open, if e > 0 is sufficiently small, then 
f e belongs to Diff 0 (T 2 ). The fixed point set of f £ is H. Moreover f £ fixes every vertical {k} x R, 
k G Z, moving every point ( k,y ), y Z,negatively in the vertical direction. 

So we can choose a point zq G {0} x (0,1) whose orbit avoids R x {1/2} and its a-limit is (0,1) and 
its w-limit is ( 0 , 0 ). 
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The point z\ = (1/2,1/2) £ dV is sent on its right by f e still on the horizontal line R x {1/2}. Let us 
choose 6 £ (0,1/2 — 1/27t) such that the orbit of Zq avoids R x [1/2 — 8 , 1/2 + <5], One can construct 

— --- OO _ 

/' £ Diff 0 (T 2 ) such that: 

• /'(z)j= S'if 8 < | \p 2 (z) - 1/21 ^ 1/2; 

• p 2 ° f'(z) = p 2 (z) for all z £ R 2 ; 

• pi o f'(z) ^ Pi(z) for all z £ R 2 ; 

• f °fe(z l) = zi + (1,0). 

Let us verify that fn = f ° / e satisfies the properties formulated in the proposition. 

The assertion (5) is satisfied by construction and (4) because the orbit of zq avoids the support of /'. 
To get (3) note that p± o f H (z) ^ p ija f e (z) > pi(z) and p 2 ° f H (z) = p 2 o ffiz) = p 2 {z) for every 
z £ int(C’). To get (2) note that p\ o f H (T) > p\ o f e {z) ^ pi(z) for every z £ V U C. In fact the last 
inequality is strict if moreover J/ZxR and we have p 2 o f H (T) = p 2 o f e (z) < p 2 {z) if z £ Z x R. 
Consequently the fixed point set of fjj is included in H. Conversely, the fact that 5 < 1/2 — 1/2-7T 
implies that H is included in the fixed point set of fn, so (1) is proved. □ 

In the same way, we can prove: 

—- --—'OO 

Proposition 12 . There exists fy £ Diff 0 (T 2 ) such that: 

( 1 ) the fixed point set of fy is V; 

(2) for every z £ H U C, one has p 2 o fy(z) ^ p 2 (z); 

(3) for every z £ int(C'), one has p 2 o fyfiz) > p 2 (z) and p\ o fyfiz) = pi(z); 

(4) there exists Zq £ R 2 such that 

lim /y( 2 'o) = ( 1 , 0 ), lim /$(%) = { 0 , 0 ); 

k — y —oo k —>-+oo 

(5) there exists z[ £ {1/2} x R such that fv(z[) = z[ + (0,1). 

It remains to prove that f = fy o f H satisfies the properties formulated in Theorem 3. Let us study 
the properties of the vector field z f(z) — z: 

• if z £ int(C) and ///(5) £ int(C), then 

Pi ° f(z) =Pi° f H (z) > pi(i) and p 2 o f(T) >p 2 o f H (z) = p 2 (z)-, 

• if z £ int(C) and /jl(?) £ V, then 

Pi ° f(z) =Pi° Jh{z) > pi(z) and p 2 o f(z) = p 2 o f H (z) = p 2 {z)\ 

• if z £ V and Jh(z) £ int(C)), then px o f(z) = pi o f H (z) > pi(z);~ 

• if z £ V and fn(z) € V, then p x o f(z) = p\ o fn(z) ^ Pi(z) and f{z) 7 ^ 0 if 2 fL 

• if S' £ H, then p 2 o f(z) ^ p 2 o /jj(5) = p 2 (z) and f{z) 7 ^ 0 if z Z 2 . 

Summarizing, the vector field z 1—7 f(fz) — z vanishes only on Z 2 and takes its value out of the negative 
cone (— 00 , 0) 2 . Moreover each vertical line {k} x R is sent on its right by / and each horizontal line 
R x {k} is sent above. One deduces that the rotation set of / is included in the non negative cone 
[0, + 00 ) 2 . Let us consider a vector v in the negative cone. The properties about vertical and horizontal 
lines are still satisfied for the diffeomorphism / — v. So the rotation set of / — v is contained in the 
non negative cone. But / — v is fixed point free, and so by a result of J. Franks [5], (0, 0) cannot be an 
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extremal point of the rotation set. Consequently, it does not belong to this set. The vector v may be 
chosen arbitrarily small, so one can perturb / in a way that (0, 0) does not belong to the rotation set. 

By construction, one knows that /(zi) = /fl-(zi) = Z\ + (1, 0) and f(z[) = fv( z i) = z o + (0,1). So the 
rotation set of / contains (0, 0), (1, 0) and (0,1) and has non empty interior. Similarly, one knows 

lim f k (z 0 ) = (0,1), lim / fc (z 0 ) = (0,0) 

AC —y — OO AC—>-+oo 

and 

lim f k (z' 0 ) = (1,0), lim f k (z' 0 ) = (0, 0). 

ac— y —oo Ac—>-+oo 

So, one can perturb / in a neighborhood of Z 2 for the C° topology and get a map /' £ Diff 0 (T 2 ) 
arbitrarily close to / such that 

f'( z l) = z i + (1, 0), f(z[) = z\ + (0,1) 
and such that there exist positive integers q and q' satisfying 

f q ( z o) = zo - (o, 1), r\z[) = z[ - (1,0). 

The rotation set of /' contains the vectors (0,1), (1,0), (—1/g, 0), (0,-1 /<?'), so its interior contains 
(0,0). 

4.4. Proof of Theorem 4. The proof is very similar to the proof of Theorem 3, writing the example 
as a composition of a “horizontal map” and a “vertical map”. We want the maps to satisfy similar 
properties but to be area preserving. The main problem is the construction of the homoclinic points, 
essential in the proof. As we will see, it is possible to do it, working in the space of homeomorphisms. 
Of course, one may ask if there exists a smooth or even a differentiable example (recall that Theorem 

[l] tells us that there is no in the space of analytic diffeomorphisms). 

We will begin by changing the definition of the set H , V and C. Fix a £ (0,1/10]. The sets 

j(z, 2 /) £ [-^, ^] 2 | \y\ < 2a|x| j , j(x,y) £ [~, ^] 2 | |x| s? 2a|t/| j 

project by n onto connected compact subsets of T 2 respectively denoted H and V. We set C = 
T 2 \ (H U V) and then define 

H = tt~ 1 (H), V = tt~\V), C = 7^- 1 (C , ). 

The analogous of Proposition El is the following: 

Proposition 13. There exists fn £ Diff 0 (T 2 ) such that: 

(1) f is area preserving 

(2) the fixed point set of fn is H; 

(3) ifz£C \ H, the vector /h^z) — 2 belongs to the cone of equation \y\ < x; 

(4) ifzGV\H and f(z) V, the vector fij(z) — z belongs to the cone of equation \y\ < x; 

(5) ifz£V\H and f(z) £ V, the vector fii(z) — z belongs to the half-plane of equation y < x; 

(6) there exists zq £ V such that 

lim /£(z 0 ) = (0,1), lim /j/(z 0 ) = (0,0); 

AC —> — OO AC—>- + 00 

(7) there exists z± £ V such that /h-(zi) = Zi + (1, 0). 
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Proof. We will construct fn step by step. Let us begin by stating elementary facts. 

There exists a diffeomorphism 9 : [0,1] —X [a, 1 — a], uniquely defined, such that, for every y G [0,1] 
the quadrilater joining the points 

(a, 1 / 2 ), ( 1 / 2 -a, 1 / 2 ), (a - a\2y - l\,y), ( 1/2 - a,y) 

and the quadrilater joining the points 

(a, 1 / 2 ), ( 1 / 2 , 1 / 2 ), (a - a\2y - l\,y), ( 1 / 2 , %)) 

have the same area. Consequently, there exists a continuous map ft:tx[0,l]->R 2 , uniquely defined, 
such that 

• the image of h is equal to (bUC)nRx [0,1]; 

• the map h is a homeomorphism between R x [0,1] and its image; 

• the map h preserves the area 

• the image of the horizontal segment 

[a - a\2y - 1|, 1 - 3a + a\y - 1/2|] x {y} 
is the broken segment passing through 

(a — a\2y — l\,y), ( 1 / 2 ,%)) and (1 - a + a\2y - 1 |, y); 

• h(x + k{l — 2a), y) = h(x, y) + (k , 0), for every ( x , j/)elx [0,1] and every k G Z; 

• the fixed point set of h is equal to V fl [—1/2, 1 / 2 ] x [0,1]. 

One gets an area preserving homeomorphism of R x [0,1] by setting 

fo(x,y) = (x + a/2 — a\y — 1/2 \,y) 

whose fixed point set is the boundary of the strip R x [0,1]. Consequently, the map h o f 0 o h~ 1 can 
be extended in a unique way to a map /i € Diff 0 (T 2 ) whose fixed point set is equal to H. Moreover 
/ is area preserving and coincides with /o on the quadrilater 

{(i,l/)eRx [0,1] | a — a\2y — l%a:^a/2 — a\y — 1/21}. 

Note that the slope of the segment joining ( a — a\2y — 1|, y) to (1/2, %)) and the slope of the segment 
joining (1/2, 9{y)) to (1 — a + a\2y — 11, j/) are both smaller than 1 in modulus. This implies that if 
z H, then fi(z) — z belongs to the cone of equation \y\ < x ; 

Now, fix/3 £ (0,1) and define a sequence (yk)kei. by the inductive relation 

Vo = 1/2, (1 - 2/fc-i) = /3(1 - yk) if % 0, y k+1 = /3y k if % 0. 

The sequence ( y k )kez is decreasing and satisfies 

lim y k = 1 , lim y k = 0 . 

k—>— oo oo 

Denote "f k the segment joining fi(0,y k ) to (0, y k +i)- The segments ( 7 fe)fegz are pairwise disjoint and 
each ff 1 ( 7 fe) is a segment disjoint from 7 ^.. Note that every vector z — z', z £ 7 k , z' G /f 1 ( 7 fc) belongs 
to the half-plane of equation y < x. So, one can construct a family of pairwise disjoint topological 
open disks {D k ) k gz such that D k is a neighborhood of 7 k disjoint from ff l (D k ) and such that every 
vector z — z ', z G D k , zJ G ff 1 {D k ) belongs to the half-plane of equation y < x. 
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For every k £ Z, one can find an area preserving lionreomorphism g^ supported on D and sending 
the point fi(0,yk) onto (0,yk+i)- There exists a unique element g of Diff 0 (T 2 ) that for every k £ Z 
coincides with g & on Dk and fixes every point with no Z 2 translation in a Dk . This homeonrorphism 
is area preserving. Note that /2 =g°fi satisfies all properties formulated in Proposition 
last one. A last step is necessary to obtain /#. 

Let us choose 1/2 < a < c < b < min(y_i, 1 — a) and a continuous map ip : [0,1] —> R. that vanishes out 
of (a, b), takes positive values in (a, b ) and satisfies p{c) = 1 — a/2 + a(c — 1/2). There exists a unique 
area preserving homeomorphism g\ £ Diff 0 (T 2 ) such that gi(x, y) = {x + <p(y),y), if (a;, y) £ R x [0,1]. 
It sends the point 

h{a /.2 - a(c - 1/2), c) = /i(a/2 - a(c - 1/2), c) = (a - a(2c - 1), c) 

onto (a/2 — a(c — 1/2) + l,c). Setting fn =gi ° /2 and z\ = (a/2 — a(c — 1/2), c), one gets a map 
that satisfies all assertions formulated in Proposition |T3} Note that the three assertions relative to the 
vector field are satisfied because the vector field z f H (z) — z is horizontal pointing to the right. □ 

Similarly we prove 

Proposition 14. There exists fv £ Diff 0 (T 2 ) such that: 

(1) f is area preserving 

(2) the fixed point set of fy is V; 

(3) ifz£C\ V, the vector fy(z) — 2 belongs to the cone of equation |x| < y; 

(4) if z £ H \ V and f(z) ^ H, the vector fvi/z) — z belongs to the cone of equation |rc| < y; 

(5) if z £ H \ V and f(z) £ H, the vector fy(z) — z belongs to the half-plane of equation x < y; 

(6) there exists z^ £ H such that 

lim /£C?o) = (1,0), lim /$(%) = { 0,0); 

K—>— OO K —>- + 00 

(7) there exists z[ £ H such that fv{z[) = z[ + (0,1). 

It remains to prove that f = fv ° fn satisfies the properties formulated in Theorem [ 4 J 
Let us prove first that each vector f(z) — z,z£ R 2 , is not a positive multiple of (—1, —1). 

• If z £ C \ H and /ll(?) £ C \ V, then /h(?) — z belongs to the cone of equation \y\ < x and 
f(z) — /ll( 2) to the cone of equation |a;| < y, so f(z) — z belongs to the half-plane of equation 
0 < x + y; 

• if z £ C\H and fn{z) € V, then f(z) — z = fii(z) — z belongs to the cone of equation \y\ < x\ 

• if z £ V \ H and /#(?) £ C\V, then fji(z) — z belongs to the cone of equation \y\ < x and 
f(z) — f H (z) to the cone of equation |cc| < y, so f(z) — z belongs to the half-plane of equation 
0 < x + y\ 

• iiz£V\H and fn(/z) G V, then f(z) — z = fij(z) — z belongs to the half-plane of equation 

y <x ; _ 

• if z £ H\ V and fv(z) ^ H, then f(z) — z = fv(z) — z belongs to the cone of equation \x\ < y, 

• if z £ H\V and fy(/z) £ H, then f(z) — z = f v (z) — z belongs to the half-plane of equation 
x< y,~ 

• if z £ H n V then f(z) = z. 
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If v is a positive multiple of (-1,-1), then / — v belongs to E>iff 0 (T 2 ), is fixed point free and area 
preserving. As said in the previous section, (0, 0) is not an extremal point of the rotation set of f — v. 
Let us prove that it does not belong to this set. 

One gets a “vertical topological line ” by adding to the broken line passing through (0,0), (1/2, a/2) 
and (0,1) all its translates by the vectors (0, k), k € Z. This line is sent to its right by the maps /i, / 2 , 
fn • The fact that V is the fixed point set of fy tells us that it is also sent to its right by f. Because 
the slopes are larger than 1 in modulus, it is strictly sent on its right by the map / — v. Similarly, one 
proves that the “horizontal topological line ” obtained by adding to the broken line passing through 
(0, 0), (1/2, a/2) and (1, 0) all its translates by the vectors (fc, 0), k G Z, is sent strictly above by / — v. 
Consequently, the rotation set of / — v is contained in [0, +oo) 2 and cannot contain (0, 0). 

By construction, one knows that f(zi) = fn(z i) = z i + (1, 0) and f(z[) = fv(z[) = z' 0 + (0,1). So the 
rotation set of / contains (0, 0), (1,0) and (0,1) and has non empty interior. Simlarly, one knows that 

lim f k (zo) = (0,1), lim f k (z 0 ) = (0,0) 

k—t—oo /c—>+oo 

and 

lim 7V 0 ) = (1,0), lim .Vo) = (0,0). 

k—> — OO Ac—>-+oo 

So, one can perturb / in a neighborhood of Z 2 for the C° topology to get an area preserving map 
f G Diff 0 (T 2 ) such that 

T(zi) =21 + (1, 0), f(z[) = z[ + (0,1) 
and such that there exist positive integers q and q' satisfying 

/Vo) = zo - (0,1), r\z[) = - (1,0). 

The rotation set of /' contains the vectors (0,1), (1,0), (— 1/q, 0), (0, —1 /q'), its interior contains (0, 0). 

Acknowledgements: We would like to thank Professor Robert Roussarie and Professor Freddy Du- 
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area-preserving diffeomorphisms of the plane. 
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